Model-based 3D Hand Tracking with on-line Hand Shape Adaptation

3D hand tracking is an interesting problem with high complexity due to the high dimensionality of the human hand and its frequent and often severe self occlusions. Most of the curent methods, track a human hand under the assumption that its parameters (e.g., finger lengths, palm dimensions, e.t.c.) are already known. This assumption limits the applicability of tracking methods. Recently, a few approaches that attempt to solve the hand shape estimation problem have been proposed [5, 6]. In this work we present an on-line method that solves simultaneously the hand tracking and hand shape estimation problems. Let xt and yt be the pose and shape state at time step t respectively. For the pose estimation the Bayesian Hierarchical Model Framework (HMF) is employed [3, 4]. The framework uses six auxiliary models that lie in lower dimensional spaces as proposals for the 26-DOF main model of the hand. The shape estimate at each time step is provided by per-frame shape parameters optimization, followed by a robust fitting framework. The per-frame optimizer generates possible shape proposals y t by optimizing the shape parameters at each frame given fixed (i.e., already estimated) pose parameters. Since the actual shape parameters are constant, the robust fitting cross-validates the shape proposals over a frame history. The output of the fitting is the best estimate given the considered history of the shape parameters ȳt that is used in the subsequent frame by the pose tracker. Hand Model The shape of the hand yt is parametrized by an 11D vector that controls finger lengths and widths and the width and height of the palm. The pose of the hand xt is parametrized by a 27D vector. The kinematics of each finger are modeled using four parameters, two for the base angles and two for the remaining joints. The global position of the hand is represented by a fixed point on the palm and the global orientation. Pose Tracking The HMF tracking framework [3, 4] that is used to track the hand pose updates at each frame t the pose parameters xt given the estimate of the shape parameters ȳt−1. The HMF uses several auxiliary models that are able to provide information for the state of the main model which is to be estimated. Each of the auxiliary models tracks a distinct part of the hand; we use one for the palm with 6-DOF for its 3D position and orientation and one for each finger with 4-DOF for the joint angles. A particle filter is used to sequentially updates the sub-states. Shape Optimization At each time step t the particle filter described above maintains a set of N weighted particles for the main model. An optimization of the shape parameters using the PSO algorithm is performed independently for the N pso << N particles with the higher weights resulting in N pso updated estimates for the shape parameters paired with the corresponding pose parameters. The likelihood of these pairs is calculated and the shape parameters with the max-likelihood y t are retained as the current shape estimate. Shape Fitting The per-frame shape estimates up to the current frame are processed by a robust fitting framework. The framework stores a history of N f frames along with their corresponding poses HF = {z f , x̄ f } N f f=1, and a history of Ns shape parameters HS = {y s }s s=1. Every shape ys in history HS is paired with every pose x̄ f in history H f . The likelihood L([x̄ f ,y pso s ],z f ) of each pair is evaluated and the shape parameters are ranked according to that likelihood. The per-frame ranks R f (x f ,ys) of each shape parameter set y s are then averaged to obtain the global rank for the set R(ys). The new estimate for the shape parameters is selected by choosing the estimate with the best average rank among the history frames. Experiments We used real data obtained by RGB-D sensors to qualitatively evaluate the methods and synthetic data for quantitative evaluations. The methods that have been included in our comparative evaluation are: (i) HMF: The method of [4] that tracks a hand without estimating its shape. (ii) SOP: Tracking the hand through HMF and perform only shape optimization per frame. (iii) SFT: The full proposed method. The synthetic dataset that we used for the evaluation consists of 1400 frames of (a) (b) (c) (d)